User interface for surfacing contextual actions in a mobile computing device

ABSTRACT

A system and method for surfacing tailored actionable items associated with applications in a user experience of a mobile computing device is disclosed. The method includes determining an event associated with a application, determining an action library including a plurality of actionable items, identifying, from the action library, one or more actionable items associated with one or more applications based on the event, generating an action menu based on the one or more actionable items, the action menu including a deep link to perform the one or more actionable items, and presenting the action menu in association with the application in response to the event.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority under 35 U.S.C. § 119 of U.S.Provisional Patent Application No. 62/528,802, filed Jul. 5, 2017, andentitled “Ultra-Portable Personal Computing and Communication Device(Companion Device) and Associated User Interfaces,” which isincorporated by reference in its entirety.

BACKGROUND Field of the Invention

The specification generally relates to surfacing tailored actionsassociated with one or more applications on a mobile computing device.In particular, the specification relates to a system and method forimproving user experience with the mobile computing device byautomatically surfacing the tailored actions in the user interface ofthe mobile computing device based on context.

Description of the Background Art

The application ecosystem of mobile computing devices has becomecomplex. There appears to be an application for everything. Furthermore,additional functionality is constantly being added to each applicationto increase user interaction with the application and functionalityprovided by the application. Typically, the operating system in mobilecomputing devices requires traversal of several layers of menus or webpages to access the functionality or data deep within the pages of anapplication. As a consequence, user experience is often cumbersome andinefficient. For example, the user has to launch the application andnavigate the application by making various selections to access adesired function or data within the application. This reduces userproductivity because the user spends more valuable time launching theapplication and traversing longer through menus and web pages to accessthe application. Additionally, it is difficult to navigate theapplication to access and initiate the desired function in a timelymanner in mobile computing devices with a small form factor. It isimportant to improve the accessibility and ease of initiating actionsassociated with the applications in mobile computing devices.

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

SUMMARY

The techniques introduced herein overcome the deficiencies andlimitations of the existing solutions, at least in part, with a systemand method for surfacing tailored actionable items associated withapplications in a user experience of a mobile computing device.

According to one innovative aspect of the subject matter described inthis disclosure, a computer-implemented method comprises: determining anevent associated with a first application; determining an action libraryincluding a plurality of actionable items; identifying, from the actionlibrary, a first actionable item associated with the first applicationbased on the event; generating a first action menu based on the firstactionable item, the first action menu including a first deep link toperform the first actionable item; and presenting the first action menuin association with the first application in response to the event.

In general, another innovative aspect of the subject matter described inthis disclosure may be embodied in systems comprising: one or moreprocessors; a memory storing instructions, which when executed cause theone or more processors to: determine an event associated with a firstapplication; determine an action library including a plurality ofactionable items; identify, from the action library, a first actionableitem associated with the first application based on the event; generatea first action menu based on the first actionable item, the first actionmenu including a first deep link to perform the first actionable item;and present the first action menu in association with the firstapplication in response to the event.

In general, another innovative aspect of the subject matter described inthis disclosure may be embodied in computer-implemented methodscomprising: receiving a signal stream from a mobile computing device;processing the signal stream; detecting an activity and a context of theactivity based on the processed signal stream; determining an actionlibrary including a plurality of actionable items associated with one ormore applications; determining an actionable item relating to theactivity and the context of the activity from the action library;generating, an action menu including a direct link to perform theactionable item; and presenting the action menu in association with thedetected activity.

In general, another innovative aspect of the subject matter described inthis disclosure may be embodied in computer-implemented methodscomprising: receiving a prolonged selection of a first application inassociation with an application launcher, determining a category of thefirst application; determining an action library including a pluralityof actionable items associated with the category; determining a firstactionable item associated with the first application from the actionlibrary; determining a second actionable item associated with a secondapplication from the action library based on an action category of thesecond actionable item matching the category of the first application;generating an action menu based on the first actionable item and thesecond actionable item, the action menu including a first deep link toperform the first actionable item and a second deep link to perform thesecond actionable item; and presenting the action menu in associationwith the first application in response to the prolonged selection of thefirst application.

Other aspects include corresponding methods, systems, apparatuses, andcomputer program products for these and other innovative aspects.

The features and advantages described herein are not all-inclusive andmany additional features and advantages will be apparent in view of thefigures and description. Moreover, it should be noted that the languageused in the specification has been principally selected for readabilityand instructional purposes and not to limit the scope of the techniquesdescribed.

BRIEF DESCRIPTION OF THE DRAWINGS

The techniques introduced herein are illustrated by way of example, andnot by way of limitation in the figures of the accompanying drawings inwhich like reference numerals are used to refer to similar elements.

FIG. 1 is a high-level block diagram illustrating one embodiment of asystem for surfacing tailored actionable items associated withapplications in a user experience of a mobile computing device.

FIG. 2A is a block diagram illustrating one embodiment of a computingdevice including an action platform application.

FIG. 2B is a block diagram illustrating an example embodiment of anaction recommender.

FIG. 3 is a block diagram illustrating another example embodiment of theaction platform application.

FIG. 4 shows example graphical representations illustrating userinterfaces for surfacing actionable items relating to an application inthe application launcher.

FIG. 5 shows example graphical representations illustrating embodimentsof user interfaces for depicting an action menu relating to anapplication.

FIG. 6 shows example graphical representations illustrating userinterfaces for editing the actions in the action menu.

FIG. 7 shows example graphical representations illustrating userinterfaces for surfacing actionable items relating to a secondapplication in the action menu of a first application.

FIG. 8 shows example graphical representations illustrating userinterfaces for surfacing actionable items relating to the notificationof an application in the user interface.

FIG. 9 shows example graphical representations illustrating userinterfaces for surfacing actionable items in response to a gestureinput.

FIG. 10 shows example graphical representations illustrating userinterfaces for surfacing actionable items in response to a contextualactivity.

FIG. 11A-11B show example graphical representations of embodiments ofthe user interface for customizing settings on the mobile computingdevice.

FIG. 12 is a flow diagram illustrating one embodiment of an examplemethod for creating an action library.

FIG. 13 is a flow diagram illustrating one embodiment of an examplemethod for generating an action menu relating to an application.

FIG. 14 is a flow diagram illustrating one embodiment of an examplemethod for surfacing actionable items relating to a second applicationin the action menu of a first application.

FIG. 15 is a flow diagram illustrating one embodiment of an examplemethod for surfacing actionable items relating to a notification of anapplication in an action menu.

FIG. 16 is a flow diagram illustrating one embodiment of an examplemethod for surfacing actionable items relating to a detected activityand associated context.

DETAILED DESCRIPTION

FIG. 1 is a high-level block diagram illustrating one embodiment of asystem 100 for surfacing tailored actionable items associated withapplications in a user experience of a mobile computing device 115. Theillustrated system 100 may have one or more mobile computing devices 115a . . . 115 n, which can be accessed by users, an action platform server110, and a plurality of web servers 101. In FIG. 1 and the remainingfigures, a letter after a reference number, e.g., “115 a,” represents areference to the element having that particular reference number. Areference number in the text without a following letter, e.g., “115,”represents a general reference to instances of the element bearing thatreference number. In the illustrated embodiment, these entities of thesystem 100 are communicatively coupled via a network 105.

The network 105 can be a conventional type, wired or wireless, and mayhave numerous different configurations including a star configuration,token ring configuration, or other configurations. Furthermore, thenetwork 105 may include a local area network (LAN), a wide area network(WAN) (e.g., the Internet), and/or other interconnected data pathsacross which multiple devices may communicate. In some embodiments, thenetwork 105 may be a peer-to-peer network. The network 105 may also becoupled to or include portions of a telecommunications network forsending data in a variety of different communication protocols. In someembodiments, the network 105 may include Bluetooth communicationnetworks or a cellular communications network for sending and receivingdata including via short messaging service (SMS), multimedia messagingservice (MMS), hypertext transfer protocol (HTTP), direct dataconnection, WAP, email, etc. Although FIG. 1 illustrates one network 105coupled to the mobile computing devices 115, the action platform server110 and the web server 101, in practice one or more networks 105 can beconnected to these entities.

In some embodiments, the system 100 includes a plurality of web servers101 coupled to the network 105 for communication with the othercomponents of the system 100, such as the plurality of mobile computingdevices 115 and the action platform server 110. A web server 101 may be,or may be implemented by, a computing device including a processor, amemory, applications, a database, and network communicationcapabilities. The web server 101 may be a Hypertext Transfer Protocol(HTTP) server, a Representational State Transfer (REST) service, orother server type, having structure and/or functionality for processingand satisfying content requests and/or receiving content from one ormore mobile computing devices 115 that are coupled to the network 105.In some embodiments, the web server 101 includes an online service 111dedicated to providing access to various services and informationresources hosted by the web server 101 via web, mobile, and/or cloudapplications on the one or more mobile computing devices 115.

The online service 111 may obtain and store user data, content items(e.g., videos, text, images, etc.), and interaction data reflecting theinteraction of users with the content items. User data, as describedherein, may include one or more of user profile information (e.g., userid, user preferences, user history, etc.), logged information (e.g.,clickstream, IP addresses, user device specific information, historicalactions, etc.), and other user specific information. In someembodiments, the online service 111 allows users to share content withother users (e.g., friends, contacts, public, similar users, etc.),purchase and/or view items (e.g., books, movies, videos, music, games,merchandise, flights, hotels, etc.), and other similar actions. Forexample, the online service 111 may be a music streaming service, a webmapping service, a multimedia messaging service, an electronic mailservice, a news service, a news aggregator service, a social network, aphoto and video-sharing social networking service, a ridesharingservice, an online banking service, a cloud storage service, an onlineinformation database service, a travel service, an online e-commercemarketplace, a ratings and review service, a restaurant-reservationservice, a food delivery service, a search service, a health and fitnessservice, home automation and security, Internet of Things (IOT), etc. Itshould be noted that the list of items provided as examples for theonline service 111 above are not exhaustive and that others arecontemplated in the techniques described herein.

In some embodiments, the web server 101 sends and receives data to andfrom other entities of the system 100 via the network 105. In theexample of FIG. 1, the components of the web server 101 are configuredto implement an application programming interface (API) 109. Forexample, the API 109 may be a software interface exposed over the HTTPprotocol by the web server 101. The API 109 includes a set ofrequirements that govern and facilitate the movement of informationbetween the components of FIG. 1. For example, the API 109 exposesinternal data and functionality of the online service 111 hosted by theweb server 101 to API requests originating from the action platformapplication 103 b implemented on the mobile computing device 115. Viathe API 109, the action platform application 103 b passes theinformation along with a set of parameters that needs servicing to aninternal function of the online service 111 and then returns an object(e.g., XML or JSON) with associated results to the mobile computingdevice 115. The web server 101 may include a database 199. The database199 may be a separate entity coupled to the web server 101 over thenetwork 105 to store structured data in a relational database and a filesystem (e.g., HDFS, NFS, etc) for unstructured or semi-structured data.In some embodiments, the web server 101 may be configured to implementthe action platform application 103 c. It should be understood that theweb server 101 and the application programming interface 109 may berepresentative of one online service provider and that there may bemultiple online service providers coupled to network 105, each havingits own server or a server cluster, applications, applicationprogramming interface, and database 199.

The action platform server 110 may be configured to implement the actionplatform application 103 a described in detail below with reference toFIGS. 2A, 2B and 3. In some embodiments, the action platform server 110is a backend hardware or software server that receives and processesinformation and analytics from the plurality of mobile computing devices115. Although, only a single action platform server 110 is shown in FIG.1, it should be understood that there may be any number of actionplatform servers 110 or a server cluster, which may be load balanced.

In some embodiments, the servers 101 and 110 may each be a hardwareserver, a software server, or a combination of software and hardware. Insome implementations, the servers 101 and 110 may each be one or morecomputing devices having data processing (e.g., at least one processor),storage (e.g., a pool of shared or unshared memory), and communicationcapabilities. For example, the servers 101 and 110 may include one ormore hardware servers, server arrays, storage devices and/or systems,etc. Also, instead of or in addition, the servers 101 and 110 may eachimplement their own API for the transmission of instructions, data,results, and other information between the servers 101 and 110 and anapplication installed or otherwise implemented on the mobile computingdevice 115. In some implementations, the servers 101 and 110 may includeone or more virtual servers, which operate in a host server environmentand access the physical hardware of the host server including, forexample, a processor, memory, storage, network interfaces, etc., via anabstraction layer (e.g., a virtual machine manager).

In some embodiments, the mobile computing device 115 may be anultra-portable computing device having a small form factor with dataprocessing and communication capabilities. The mobile computing device115 may include a memory, a processor, a camera, a communication unitcapable of accessing the network 105, a power source, and/or othersoftware and/or hardware components, such as a display (for viewinginformation provided by the web server 101 via API 109), graphicsprocessor unit (for handling general graphics and multimediaprocessing), sensor(s), sensor hub, firmware, operating systems,drivers, a subscriber identification module (SIM) or other integratedcircuit to support cellular communication, and various physicalconnection interfaces (e.g., USB, USB-C, USB Micro, etc.). In someembodiments, the mobile computing device 115 may be a companion deviceto other primary computing devices (not shown), for example a laptopcomputer, a desktop computer, a tablet computer, a mobile telephone, asmartphone, a personal digital assistant (PDA), a mobile email device, aportable game player, a portable music player, a television with one ormore processors embedded therein or coupled thereto, a webcam, a userwearable computing device or any other electronic device capable ofaccessing the network 105, etc.

In some embodiments, the mobile computing device 115 may synchronize itsdata (e.g., emails, text messages, calendar, documents, address book,files, etc.) over the network 105 with the other primary computingdevices to which it is a companion device. In some embodiments, themobile computing device 115 may be the primary computing device to thesame or different type of companion device. The mobile computing device115 may include a browser application through which the mobile computingdevice 115 interacts with the online service 111 hosted by the webserver 101, an application (e.g., HTML5 based mobile application)installed enabling the mobile computing device 115 to couple andinteract with the online service 111 hosted by the web server 101, ormay couple with the web server 101 in some other way. In someembodiments, a single user may use more than one mobile computing device115, which the action platform server 110 may track and accordinglyprovide the functionality as described herein. For example, the actionplatform server 110 may track the behavior of a user across multiplemobile computing devices 115. While FIG. 1 illustrates two mobilecomputing devices 115 a and 115 n, the disclosure applies to a systemarchitecture having one or more mobile computing devices 115. Inaddition, the mobile computing devices 115 a and 115 n may be the sameor different types of mobile computing devices.

The action platform application 103 may include software and/or logic toprovide the functionality for surfacing tailored actionable itemsrelated to one or more applications in a user experience of the mobilecomputing device 115 based on context. In some embodiments, the actionplatform application 103 can be implemented using programmable orspecialized hardware, such as a field-programmable gate array (FPGA) oran application-specific integrated circuit (ASIC). In some embodiments,the action platform application 103 can be implemented using acombination of hardware and software. In other embodiments, the actionplatform application 103 may be stored and executed on a combination ofthe mobile computing devices 115 and the action platform server 110, orby any one of the mobile computing devices 115 or action platform server110.

As depicted in FIG. 1, the action platform application 103 a and 103 bis shown in dotted lines to indicate that the operations performed bythe action platform application 103 a and 103 b as described herein canbe performed at the mobile computing device 115, the action platformserver 110, or any combinations of these components. In someembodiments, each instance 103 a and 103 b may comprise one or morecomponents of the action platform application 103 depicted in FIGS. 2Aand 2B, and may be configured to fully or partially perform thefunctionalities described herein depending on where the instanceresides. Additional structure, acts, and/or functionality of the actionplatform application 103 is described in further detail below withrespect to at least FIGS. 2A, 2B and 3. While the action platformapplication 103 is described below as a stand-alone action platformapplication, in some embodiments, the action platform application 103may be part of other applications in operation on the mobile computingdevice 115 and the action platform server 110.

In some embodiments, the action platform application 103 b may be athin-client application with some functionality executed on the mobilecomputing device 115 and additional functionality executed on the actionplatform server 110 by the action platform application 103 a. Forexample, the action platform application 103 b on the mobile computingdevice 115 could include software and/or logic for archiving a historyof user interactions with one or more applications loaded on the mobilecomputing device 115, for example on a daily basis, determiningactionable items associated with the one or more applications, forexample commonly accessed functionalities, building an action libraryincluding a plurality of actionable items of the one or moreapplications, tailoring the actionable items and surfacing the tailoredactionable items in the user experience of the mobile computing device115 on-the-go based on context. In another example, the action platformapplication 103 a on the web server 101 could include software and/orlogic for preparing learning data and training one or more action modelsusing the learning data. The operation of the action platformapplication 103 and the functions listed above are described below inmore detail below with reference to FIGS. 2-16.

FIG. 2A is a block diagram illustrating one embodiment of a computingdevice 200 including an action platform application 103. The computingdevice 200 may also include a processor 235, a memory 237, acommunication unit 241, data storage 243, an optional capture device245, input/output device(s) 247, and sensor(s) 249 according to someexamples. The memory 237 may include one or more of the operating system107 and the action platform application 103. The components of thecomputing device 200 are communicatively coupled by a bus 220. In someembodiments, the computing device 200 may be representative of themobile computing device 115, the action platform server 110, the webserver 101 or a combination of the mobile computing device 115, the webserver 101 and the action platform server 110. In such embodiments wherethe computing device 200 is representative of the mobile computingdevice 115, the web server 101 or the action platform server 110, itshould be understood that the mobile computing device 115, the webserver 101 and the action platform server 110 may include othercomponents described above but not shown in FIG. 2A.

The processor 235 may execute software instructions by performingvarious input/output, logical, and/or mathematical operations. Theprocessor 235 may have various computing architectures to process datasignals including, for example, a complex instruction set computer(CISC) architecture, a reduced instruction set computer (RISC)architecture, and/or an architecture implementing a combination ofinstruction sets. The processor 235 may be physical and/or virtual andmay include a single processing unit or a plurality of processing unitsand/or cores. In some implementations, the processor 235 may be capableof generating and providing electronic display signals to a displaydevice, supporting the display of images, capturing and transmittingimages, performing complex tasks including various types of featureextraction and sampling, etc. In some implementations, the processor 235may be coupled to the memory 237 via the bus 220 to access data andinstructions therefrom and store data therein. The bus 220 may couplethe processor 235 to the other components of the computing device 200including, for example, the memory 237, the communication unit 241, theaction platform application 103, and the data storage 243. In someimplementations, the processor 235 may be coupled to a low-powersecondary processor (e.g., sensor hub) included on the same integratedcircuit or on a separate integrated circuit. This secondary processormay be dedicated to performing low-level computation at low power. Forexample, the secondary processor can perform step counting, sensorfusion, sensor batching, etc. in accordance with the instructionsreceived from the action platform application 103.

The memory 237 may store and provide access to data for the othercomponents of the computing device 200. The memory 237 may be includedin a single computing device or distributed among a plurality ofcomputing devices as discussed elsewhere herein. In someimplementations, the memory 237 may store instructions and/or data thatmay be executed by the processor 235. The instructions and/or data mayinclude code for performing the techniques described herein. The memory237 is also capable of storing other instructions and data, including,for example, an operating system, hardware drivers, other softwareapplications, databases, etc. The memory 237 may be coupled to the bus220 for communication with the processor 235, the communication unit241, the data storage 243 and/or the other components of the computingdevice 200.

The memory 237 may include one or more non-transitory computer-usable(e.g., readable, writeable) medium, which can be any tangiblenon-transitory apparatus or device that can contain, store, communicate,propagate or transport instructions, data, computer programs, software,code, routines, etc., for processing by or in connection with theprocessor 235. In some implementations, the memory 237 may include oneor more of volatile memory and non-volatile memory. For example, thememory 237 may include, but is not limited to, one or more of a staticrandom access memory (SRAM) device, a dynamic random access memory(DRAM) device, an embedded memory device, a discrete memory device(e.g., a PROM, FPROM, ROM), a hard disk drive, an optical disk drive(CD, DVD, Blu-ray™, etc.). It should be understood that the memory 237may be a single device or may include multiple types of devices andconfigurations.

The bus 220 may represent one or more buses including an industrystandard architecture (ISA) bus, a peripheral component interconnect(PCI) bus, a universal serial bus (USB), or some other bus providingsimilar functionality. The bus 220 can include a communication bus fortransferring data between components of the computing device 200 orbetween the computing device 200 and other components of the system viathe network 105 or portions thereof, a processor mesh, a combinationthereof, etc. In some implementations, the action platform application103 and the operating system 107 may cooperate and communicate via asoftware communication mechanism implemented in association with the bus220. The software communication mechanism can include and/or facilitate,for example, inter-process communication, local function or procedurecalls, remote procedure calls, network-based communication, securecommunication, etc.

The communication unit 241 is hardware for receiving and transmittingdata by linking the processor 235 to the network 105 and otherprocessing systems. The communication unit 241 receives data such asrequests from the mobile computing device 115 and transmits the requeststo the web server 101. The communication unit 241 also transmitsinformation to the mobile computing device 115 for display, for example,in response to processing the requests. The communication unit 241 iscoupled to the bus 220. The communication unit 241 may include one ormore interface devices for wired and wireless connectivity with thenetwork 105 and the other entities and/or components of the system 100including, for example, the web server 101, the mobile computing devices115, and the action platform server 110. For instance, the communicationunit 241 may include, but is not limited to, cable interfaces (e.g.,CAT-5); wireless transceivers for sending and receiving data usingWi-Fi™; Bluetooth®, Near-field communication (NFC), cellularcommunications, etc.; universal serial bus (USB) interfaces; variouscombinations thereof; etc. The communication unit 241 may be coupled tothe network 105 via the signal line 104.

In another embodiment, the communication unit 241 may include a cellularcommunications transceiver for sending and receiving data over acellular communications network such as via short messaging service(SMS), multimedia messaging service (MMS), hypertext transfer protocol(HTTP), direct data connection, WAP, e-mail or another suitable type ofelectronic communication. The communication unit 241 may also provideother conventional connections to the network 105 for distribution offiles and/or media objects using standard network protocols such asTCP/IP, HTTP, HTTPS, and SMTP or any combination thereof.

The data storage 243 is a non-transitory memory that stores data forproviding the functionality described herein. The data storage 243 maybe a dynamic random access memory (DRAM) device, a static random accessmemory (SRAM) device, flash memory, or some other memory devices. Insome embodiments, the data storage 243 also may include a non-volatilememory or similar permanent storage device and media including a harddisk drive, a floppy disk drive, a CD-ROM device, a DVD-ROM device, aDVD-RAM device, a DVD-RW device, a flash memory device, or some othermass storage device for storing information on a more permanent basis.

The data storage 243 is communicatively coupled to the bus 220. In someembodiments, the data storage 243 may be coupled to the components 235,237, 241, 245, or 249 of the computing device 200 via the bus 214 toreceive and provide access to data. In some embodiments, the datastorage 243 may store data received from the other entities 101, 110, or115 of the system 100, and provide data access to these entities. Insome embodiments, the data storage 243 may be incorporated with thememory 237 or may be distinct therefrom. In some implementations, thedata storage 243 may include a database management system (DBMS). Forexample, the DBMS could include a structured query language (SQL) DBMS,a NoSQL DMBS, various combinations thereof, etc. In some instances, theDBMS may store data in multi-dimensional tables comprised of rows andcolumns, and manipulate, e.g., insert, query, update and/or delete, rowsof data using programmatic operations. The data stored in the datastorage 243 is described below in more detail.

The capture device 245 may be operable to capture an image or datadigitally of an object of interest. For example, the capture device 245may be a high definition (HD) camera, a regular 2D camera, amulti-spectral camera, a structured light 3D camera, a time-of-flight 3Dcamera, a stereo camera, a standard smartphone camera, a wearablecomputing device, a barcode reader, etc. The capture device 245 iscoupled to the bus to provide the images and other processed metadata tothe processor 235, the memory 237, or the data storage 243. It should benoted that the capture device 245 is shown in FIG. 2A with dashed linesto indicate it is optional. For example, where the computing device 200is the action platform server 110, the capture device 245 may not bepart of the system, where the computing device 200 is the mobilecomputing device 115, the capture device 245 is included and is used toprovide images and other metadata information described below.

The input/output (I/O) device(s) 247 may include any standard device forinputting or outputting information and may be coupled to the computingdevice 200 either directly or through intervening I/O controllers.Non-limiting example I/O devices 247 include a touch screen or any othersimilarly equipped display device equipped to display user interfaces,electronic images, and data as described herein, a touchpad, a scanner,a stylus, light emitting diode (LED) indicators, an audio reproductiondevice (e.g., speaker), a microphone, a barcode reader, an eye gazetracker, a sip-and-puff device, and any other I/O components forfacilitating communication and/or interaction with users. In differentembodiments, the display can be binary (only two different values forpixels), monochrome (multiple shades of one color), or multiple colorsand shades.

The sensor(s) 249 includes any type of sensors suitable for thecomputing device 200. The sensor(s) 249 are communicatively coupled tothe bus 220. In the context of the mobile computing device 115, thesensor(s) 249 may be configured to collect any type of signal datasuitable to determine characteristics of the mobile computing device 115and/or its internal and external environments. Non-limiting examples ofthe sensor(s) 249 include various optical sensors (CCD, CMOS, 2D, 3D,light detection and ranging (LIDAR), cameras, etc.), audio sensors,motion detection sensors, magnetometer, barometers, altimeters,thermocouples, moisture sensors, infrared (IR) sensors, radar sensors,other photo sensors, gyroscopes, accelerometers, geo-location sensors,orientation sensor, wireless transceivers (e.g., cellular, WiFi™,near-field, etc.), sonar sensors, ultrasonic sensors, touch sensors,proximity sensors, distance sensors, microphones, etc. In someembodiments, one or more sensors 249 may include externally facingsensors provided at the front side, rear side, right side, and/or leftside of the mobile computing device 115 in order to capture thesituational context surrounding the mobile computing device 115. In someembodiments, the sensor(s) 249 may include one or more image sensors(e.g., optical sensors) configured to record images including videoimages and still images, may record frames of a video stream using anyapplicable frame rate, and may encode and/or process the video and stillimages captured using any applicable methods. In some embodiments, theimage sensor(s) 249 can capture images of surrounding environmentswithin their sensor range. For example, in the context of a mobilecomputing device 115, the image sensors 249 can capture the environmentaround the mobile computing device 115 including people, ambient light(e.g., day or night time), etc.

It should be understood that other processors, operating systems,sensors, displays, and physical configurations are possible.

As depicted in FIG. 2A, the memory 237 may include the operating system107, and the action platform application 103.

The operating system 107, stored on memory 237 and configured to beexecuted by the processor 235, is a component of system software thatmanages hardware and software resources in the computing device 200. Theoperating system 107 includes a kernel that controls the execution ofthe action platform application 103 by managing input/output requestsfrom the action platform application 103. The action platformapplication 103 requests a service from the kernel of the operatingsystem 107 through system calls. In addition, the operating system 107may provide scheduling, data management, memory management,communication control and other related services. For example, theoperating system 107 is responsible for recognizing input from a touchscreen, sending output to a display screen, tracking files on the datastorage 243, and controlling peripheral devices (e.g., Bluetooth®headphones, earbuds, etc.). In one embodiment, the operating system 107is a general-purpose operating system. For example, the operating system107 may be Microsoft Windows®, Mac OS® or UNIX® based operating system.Or the operating system 107 may be a mobile operating system, such asAndroid®, iOS® or Tizen™. In another embodiment, the operating system107 may be a special-purpose operating system. The operating system 107may include other utility software or system software to configure andmaintain the computing device 200.

The action platform application 103 includes an archiver 202, an actionlibrary curator 204, a processing engine 206, a context determiner 208,an action recommender 210, and a user interface 212. The components 202,204, 206, 208, 210, and 212 of the action platform application 103 arecoupled for communication with each other and the other components 235,241, 243, 245, 247, and 249 of the computing device 200 by the bus 220.The components 202, 204, 206, 208, 210, and 212 of the action platformapplication 103 are also coupled to the network 105 via thecommunication unit 241 for communication with the other entities 101,110, and 115 of the system 100.

In some embodiments, the components 202, 204, 206, 208, 210, and 212 ofthe action platform application 103 may each include software and/orlogic to provide their respective functionality. In some embodiments,the components 202, 204, 206, 208, 210, and 212 of the action platformapplication 103 can each be implemented using programmable orspecialized hardware including a field-programmable gate array (FPGA) oran application-specific integrated circuit (ASIC). In some embodiments,the components 202, 204, 206, 208, 210, and 212 of the action platformapplication 103 can each be implemented using a combination of hardwareand software executable by the processor 235. In some embodiments, thecomponents 202, 204, 206, 208, 210, and 212 of the action platformapplication 103 may each be stored in the memory 237 and be accessibleand executable by the processor 235 to provide their respective actsand/or functionality. In any of these embodiments, the components of theaction platform application 103 may each be adapted for cooperation andcommunication with the processor 235, the memory 237, and othercomponents of the computing device 200 via the bus 220.

The archiver 202 may include software and/or logic to provide thefunctionality for determining and archiving interaction of users withone or more applications on the mobile computing device 115. In someembodiments, the archiver 202 determines a frequency of user accessingone or more applications in an application launcher of the mobilecomputing device 115. For example, the archiver 202 determines a numberof times a user launched a messaging application in a day, week, month,etc. The archiver 202 determines a time during which the user accessesthe one or more applications. For example, the archiver 202 determinesthat the user opens an email application in the morning between 8 AM and10 AM. The archiver 202 determines a location where the user accessesthe one or more applications. For example, the archiver 202 determinesthat the user launches a Wi-Fi home automation application when the userconnects to a Wi-Fi spot at home. In another example, the archiver 202determines that the user launches a calendaring application when theuser is in the office. In yet another example, the archiver 202determines that the user launches a web mapping service when the user iscommuting. The archiver 202 determines a set of parameters of dataand/or functions accessed within the one or more applications by theusers. For example, the archiver 202 determines that a user launches amusic streaming application and commonly plays a “Top 100 Billboard”playlist. In another example, the archiver 202 determines that the userlaunches the messaging application and commonly sends a text to aparent. In yet another example, the archiver 202 determines that theuser opens the phone application and commonly calls a spouse. Thearchiver 202 determines interactions of the user with the applicationsin the application launcher. For example, the archiver 202 determines aselection of application icon, reordering of application icons in theapplication launcher, dismissal of notifications from applications,removing of application icons from the application launcher, groupingapplication icons in a folder, etc. The archiver 202 determines a userprofile account of the user on the one or more applications. Forexample, the archiver 202 determines user preferences and settings basedon accessing the user profile maintained by the online service 111 inthe web server 101. The archiver 202 combines one or more of the abovedescribed determinations to archive new set of interaction data. Forexample, the archiver 202 determines that the user uses the phoneapplication commonly to call a spouse when the user is commuting to homefrom the office. Example parameters of data archived by the archiver 202from the user interaction with one or more applications include but notlimited to frequently accessed phone contact, frequently accessed socialnetwork friend, frequently accessed email recipient, frequently accessedtext message recipient, favorite playlist, recently accessed playlist,work address, home address, daily exercise routine, most commonly usedpayment method, etc. or any combination thereof in one or moreapplications. Example parameters of function archived by the archiver202 from the user interaction with one or more applications include butnot limited to streaming a playlist, checking traffic conditions,getting map directions, requesting a rideshare service, booking arestaurant table, making a phone call, sending a text message, sharing aphoto, posting a status, performing a social check-in to a physicallocation, recording an exercise log, paying with rewards card, accessingcar keys, etc. or any combination thereof in one or more applications.In some embodiments, the archiver 202 stores the interaction data ofusers with one or more applications in the archive data 222 of the datastorage 243.

The action library curator 204 may include software and/or logic toprovide the functionality for creating and managing an action library224 in the data storage 243. The action library 224 includes a pluralityof actionable items associated with the one or more applications on themobile computing device 115. The action library curator 204 determineswhether an API 109 is available for the one or more applications on themobile computing device 115. For example, the action library curator 204pings the online service 11 hosted by the web server 101 and uses theAPI 109 to identify the accessible application functionalities and/ordata. The API 109 can be an internal or external API depending on theconfiguration of the online service 111 hosted by the web server 101.The action library curator 204 determines a function of the applicationaccessible via the API 109. The action library curator 204 uses an APIkey to get access to the API 109 of the online service 111. For example,the API key may be available based on a subscription or partnership withthe online service 111 and provides the action library curator 204 witha read access, write access, or both to the API 109 depending on theconfiguration of the API 109. The action library curator 204 uses theAPI 109 to parse an application and determine one or more constituentfunctions, tasks, or activities associated with the applicationavailable via the API 109. For example, an API 109 associated with anemail online service 111 exposes a function to draft a new email. Inanother example, an API 109 associated with a social networking service111 exposes a function to update a user's social media status. Theaction library curator 204 determines an API call to perform or initiatethe function of the application via the API 109. The API call can beused to submit a request to the API 109 to perform a function of theapplication. The request includes the parameters or data associated withperforming the function. The API 109 services the request of the APIcall and returns a response. The action library curator 204 generates anactionable item using the API call and one or more parameters associatedwith the application. The actionable item of an application may be auser interface element that links directly to an API call toautomatically transmit a request to the API 109 of the online service111 for performing a function, task or activity associated with theapplication. The actionable item performs the function of theapplication without requiring a launch of the associated application onthe mobile computing device 115. In some embodiments, the actionableitem may be a short cut to a specific function in the application. Theaction library curator 204 generates the actionable item using a uniformresource identifier (URI) of the function of the application. Theactionable item uses the URI to provide a deep link that links to aspecific location or page associated with the function of theapplication and initiates the function without any prompts, interstitialpages, or logins. For example, the action library curator 204 creates anactionable item “Email Rob” that can be invoked for initiating an emailmessage to send to a contact named Rob without having to perform asequence of steps including launching the associated email applicationin an application launcher, searching for the contact named Rob in thecontact list, selecting the contact, and selecting the option to draftan email. In another example, the action library curator 204 creates anactionable item “Rock Playlist” that can be invoked for opening a songplaylist of ‘Rock’ genre without having to perform a sequence of stepsincluding launching the associated music streaming application,selecting a library in the start page, selecting a playlist option inthe second page, browsing the listed playlists to select the “RockPlaylist” in the third page, and selecting the “Play” option in thefourth page of the music streaming application. In some embodiments, theaction library curator 204 generates actionable items related to thesystem (e.g., system actions) of the application. The system actionsinclude view application information, move application icon command,uninstall command, etc. The action library curator 204 classifies acategory of the actionable item. For example, the action library curator204 identifies actionable items for one or more social networkingapplications and groups the identified actionable items under a ‘social’category. In some embodiments, the action library curator 204 parses thearchive data 222 in the data storage 222, identifies the one or moreapplications and parameters associated with the interaction data, andbuilds the action library 224 based on the interaction data in thearchive data 222 as described above.

The processing engine 206 may include software and/or logic to providethe functionality for processing information including one or more ofthe sensor data associated with the sensor(s) 249 and the interactiondata of users with applications from the archiver 202. The sensor(s) 249associated with the computing device 200 generate the sensor data. Insome embodiments, the processing engine 206 receives the sensor datafrom the sensor(s) 249 and processes the sensor data. Example processeddata includes, but not limited to, data describing a number of steps,data describing a geographic location (e.g., a latitude, a longitude andan elevation of a location) and velocity from a GPS sensor, datadescribing a presence of other devices in proximity to the mobilecomputing device 115, data describing a movement and a direction ofmovement from an accelerometer, data describing brightness in anenvironment from a light detector, data describing detecting ambientsounds from a microphone, data describing detecting wireless accesspoints from a wireless transceiver, etc. Other example processed data ispossible. In some embodiments, the processing engine 206 receives userinteraction data including but not limited to installed applications onthe mobile computing device 115, user interaction data with applications(e.g., playing music, composing email message, posting a photo to socialnetwork, etc.), application usage history data, application notificationdata, communication and call history data, etc. from the archiver 202Other example user interaction data is possible. The processing engine206 receives the sensor data and user interaction data with permissionfrom the user. In some embodiments, the processing engine 206 combinesthe sensor data and user interaction data into one signal stream forjoint processing. The processing engine 206 filters the sensor dataand/or the user interaction data (e.g., signal stream) to determine anevent. An event may be an action or occurrence relating to anapplication on a user interface of the mobile computing device 115. Forexample, the processing engine 206 may determine a graphical userinterface (GUI) event. A source of GUI events includes the user, who mayinteract with the applications in an application launcher by way of, forexample, button presses, swipes, a prolonged selection of an applicationicon (e.g., long-press), a selection of a notification related to anapplication, a gesture, etc. An event may also be a change in anenvironment of the mobile computing device 115. For example, theprocessing unit 206 may determine a change in location coordinates andtime and outputs a filtered signal data including geo-location of themobile computing device 115.

The context determiner 208 may include software and/or logic to providethe functionality for receiving the processed data from the processingengine 206 and determining a context in the processed data. The contextdeterminer 208 analyzes the processed data and determines a contextassociated with the processed data. In some embodiments, the contextdeterminer 208 uses the processed data as input to activity recognitionalgorithms to determine a contextual user activity. For example, thecontext determiner 208 determines a user activity such as, running,walking, commuting, etc. based on analyzing the processed data. In someembodiments, the context determiner 208 determines whether the contextis ongoing and persisting for a threshold period of time. If the contextis determined to persist, the context determiner 208 sends an indicationof the context to the action recommender 210. For example, if thecontext determiner 208 determines a context of the user of the mobilecomputing device 115 is that the user is starting a run and the contextdeterminer 208 detects the running activity is continuous for athreshold period of 30 seconds, then the context determiner 208 sends anindication of the context of running to the action recommender 210. Thecontext determiner 208 combines different types of data in the processeddata to determine a context in terms of human understandable actions.For example, the context determiner 208 combines one or more of thefollowing data including: (1) a change in geo-location data from the GPSsensor, (2) proximity detection data indicating presence of anautomobile (e.g., Bluetooth enabled), and (3) a scheduled business eventin a calendaring application to determine that the user is commuting ina car to a business meeting. In another example, the context determiner208 combines one or more of the following data including: (1) a time,(2) geo-location data from a GPS sensor, and (3) a detection of an openWi-Fi access point to determine that the user is inside a coffeehouse.In a third example, the context determiner 208 combines one or more ofthe following data including: (1) motion data from an accelerometer, (2)ambient illumination data from a light sensor, (3) geo-location data,and (4) energy usage data from a power detector on the mobile computingdevice 115 to determine that the user is running. The context determiner208 sends information including the determined context to the actionrecommender 210.

The action recommender 210 may include software and/or logic to providethe functionality for receiving the context from the context determiner208 and surfacing one or more actionable items from the action library224 based on the context. For example, the actionable items are surfacedin the user experience of the mobile computing device 115.

FIG. 2B is a block diagram illustrating an example embodiment of anaction recommender 210. As depicted, the action recommender 210 mayinclude an action model generator 302, a launcher action generator 304,a notification action generator 306, a contextual action generator 308,and an action resolver 310.

The action model generator 302 receives a training dataset includingclassified, labeled features suitable for training a machine learningmodel. The training dataset may correspond to a particular demographicgroup to which the user of the mobile computing device 115 belongs. Thetraining dataset includes context signals as features and an actionableitem as the label. The action model generator 302 generates one or moreaction models based on the training dataset. The action model generator302 stores the action model(s) 226 in the data storage 243. The actionmodel generator 302 may include a machine learning logic to train theone or more action models. In some embodiments, the action modelgenerator 302 may use one or more machine learning models to create andtrain the one or more action models on the training data. Depending onthe embodiment, the one or more machine learning models used may includesupervised machine learning models only, unsupervised machine learningmodels only, or both supervised and unsupervised machine learningmodels. Example machine learning models include but are not limited to aclassification model, a regression model, a ranking model, asemi-supervised model, a density estimation model, a clustering model, adimensionality reduction model, a multidimensional querying model, anensemble model, convolutional neural network, etc. The action modelgenerator 302 provides the trained action models to the launcher actiongenerator 304, the notification action generator 306, and the contextualaction generator 308. The trained action models provided to thecomponents 304, 306, and 308 may be the same or different models. Insome embodiments, the components 304, 306, and 308 may use the one ormore action models to surface actionable items from the action library224. The action model may receive as input the contextual state of themobile computing device 115 and generate a list of actionable items asdescribed herein. The action model generator 302 retrieves interactiondata of user from the archive data 222 and updates the training dataset.The action model generator 302 retrains the action model(s) 226 in thedata storage 243 using the updated training dataset. Other systemarchitectures for implementing machine learning logic with various typesof supervised learning algorithm and/or unsupervised learning algorithmare also possible and contemplated.

The launcher action generator 304 determines the context is associatedwith an application launcher of the mobile computing device 115 andsurfaces one or more actionable items from the action library 224 inresponse to the context. For example, the launcher action generator 304determines the context is one of accessing actions associated with aselect application by long-pressing the select application in theapplication launcher. The launcher action generator 304 generates anaction menu including one or more actionable items related to theapplication. The actionable items related to one or more applicationscan be defined as tailored actionable items. For example, the tailoredactionable items related to an application are quick actionable itemsmaking accessible to the user the most commonly accessed data and/orfunctions of the application. The launcher action generator 304populates the action menu with the actionable items of the applicationand ranks the actionable items in the action menu based on a rankingscheme. For example, the actionable items related to the application inthe action menu may be ranked such that the most commonly accessed dataand/or functions of the application are at the top of the action menu.In some embodiments, the launcher action generator 304 cross-pollinatesthe action menu with actionable items of another application. Forexample, the launcher action generator 304 determines a category of afirst application for which action menu is being generated. The launcheraction generator 304 uses the category to search the action library 224for actionable items. The launcher action generator 304 identifiesactionable items of a second application that belongs to the samecategory as the first application. The launcher action generator 304generates the action menu including the actionable items of the firstand the second application.

The notification action generator 306 determines a context is associatedwith a notification of an application and surfaces one or moreactionable items from the action library 224 in response to the context.For example, the notification action generator 306 determines thecontext is opening a notification of an application by swiping down thenotification in the notification bar for a detailed view. Thenotification action generator 306 generates an action menu includingactionable items of applications related to the context of thenotification. For example, a notification from a music streamingapplication may indicate a new album by an artist is now available forpurchase. The notification action generator 306 generates an action menuwith a first actionable item to add the track to a playlist on the musicstreaming application, a second actionable item to buy the album on anonline digital media store application, and a third actionable item tofind tickets to a concert of the artist on an online ticket exchangeservice application. The first, second and third actionable items aretailored actionable items surfaced in the action menu to facilitate theuser to take an action in relation to the notification about the newalbum.

The contextual action generator 308 determines the context is associatedwith an ambient environmental conditions of the mobile computing device115 and surfaces one or more actionable items from the action library224 in response to the context. For example, the contextual actiongenerator 308 determines the context is associated with a fitnessactivity and generates an action menu or contextual experience menu thatincludes one or more actionable items related to fitness surfaced fromthe action library 224. The example actionable items may prompt the userto record the fitness activity with a fitness application, start playinga workout playlist on a music streaming application, etc. As such, theactionable items are tailored because they match the context of thefitness activity. In another example, the contextual action generator308 determines the context of the mobile computing device 115 gettingconnected to home Wi-Fi network is indicative of the user getting intothe house. The contextual action generator 308 generates an action menuincluding one or more actionable items related to the home context wherethe actionable items prompt the user to turn on lights, play music on asmart speaker, adjust thermostat, etc. In some embodiments, thecontextual action generator 308 determines the context is associatedwith receiving a gesture input from the user and surfaces contextualactions including one or more actionable items from the action libraryto satisfy the gesture input. For example, the gesture input can be astroke of a letter on a gesture pad of the mobile computing device 115.The contextual action generator 308 searches for actionable itemsrelating to the letter from the action library and populates an actionmenu with the actionable items. Examples of gesture input include butnot limited to letters, numbers, emoticons, doodles, etc.

The action resolver 310 receives a user selection of an actionable itemin an action menu and uses the API 109 of the online service 111 hostedby the web server 101 to perform the action of an application associatedwith the actionable item without launching the application on the mobilecomputing device 115. The action resolver 310 invokes the API callassociated with the actionable item when the actionable item is selectedin the user interface. The API call passes the parameters of theactionable item to the API 109 to execute the function of theapplication. In some embodiments, the action resolver 310 generates asecondary action screen in response to receiving the user selection ofthe actionable item. The actionable item may need a multi-step action.The secondary action screen is generated as a HTML5 client to receiveadditional user input to complete the action of the applicationassociated with the actionable item. For example, an actionable item tosend a text message to Alice needs two user inputs to complete theaction: one to select the actionable item in the action menu and thesecond to enter the message in a secondary action screen. The actionresolver 310 transmits the text message entered in the secondary actionscreen to the API 109 of the text messaging application using the APIcall and the text message is sent to Alice without the user needing tolaunch the text messaging application.

Referring back to FIG. 2A, the user interface engine 212 may includesoftware and/or logic to provide the functionality for generating andtransmitting user interfaces to the display of the mobile computingdevice 115. In some embodiments, the user interface engine 212 receivesinstructions from the action recommender 210 to generate a userinterface presenting an action menu including one or more actionableitems. The user interface engine 212 sends instructions includinggraphical user interface data to the mobile computing device 115 via thecommunication unit 241 causing the mobile computing device 115 todisplay the data as a graphical user interface.

Referring now to FIG. 3, another example implementation of the actionplatform application 103 is shown. FIG. 3 shown the general data flowthrough the action platform application 103 to surface the actionableitems of one or more applications in the user experience of the mobilecomputing device 115. FIG. 3 illustrates how components of the actionplatform application 103 communicate with each other and the web servers101. In particular, the action library curator 204 and the actionrecommender 210 cooperate with the APIs 109 of the online services 111hosted by multiple web servers 101 via the signal lines 351 and 353respectively. The API 109 exposes the functionality and data of theonline service 111 to the action platform application 103 for the actionlibrary curator 204 to create the action library 224 and for the actionrecommender 210 to pass parameters of the invoked actionable items tothe API 109 to perform a function of an associated application. Theaction recommender 210 surfaces the actions from the action library 224for initiating the actions of the applications without launching theapplications on the mobile computing device 115.

The archiver 202 archives how the user of the mobile computing device115 interacts with the applications accessible on the mobile computingdevice 115 and creates a profile of the user based on the interactiondata. The archiver 202 may be coupled to the mobile computing device 115via signal line 369 to receive the interaction data from the mobilecomputing device 115. The archiver 202 identifies and groups theapplications associated with the mobile computing device 115 under oneor more categories. For example, the archiver 202 groups theapplications, such as text messaging, email applications, chatapplications, etc. under ‘Communication’ category. The archiver 202archives how the data or result of one application is used to interactwith another application by the user. For example, when a notificationfrom a calendaring application includes a reminder about a scheduledevent 30 minutes in the future, the calendaring application often doesnot receive any further action from the user. However, the reminderwould cause the user to launch a web mapping application to determinedirections to the event location or traffic conditions, booking a carvia the ride sharing application, or initiating a phone call to theevent host. In some embodiments, once such interaction data isdetermined, the archiver 202 transmits the data to the data storage 243for storing in the archive data 222. The archiver 202 may be coupled tothe data storage 243 via signal line 357. The archiver 202 alsotransmits the interaction data to the action library curator 204. Thearchiver 202 may be coupled to the action library curator 204 via signalline 371.

Turning now to the action library curator 204, the action librarycurator 204 receives the interaction data from the archiver 202. Theaction library curator 204 parses the interaction data, generates alibrary of actionable items and provides them to the action recommender210 via the signal line 373 as has been described above. The actionlibrary curator 204 also transmits the library for storing in the actionlibrary 224 of the data storage 243. The action library curator 204 maybe coupled to the data storage 243 via signal line 359. Examples ofaction library categories include but not limited to music actionlibrary (e.g., play ‘Workout Mix,’ Launch ‘Radio X City,’ stream‘Background Noise,’ etc.), auto action library (e.g., start engine, opentrunk, unlock car, etc.), productivity action library (e.g., createevent, set reminder, set alarm, etc.), fitness action library (e.g.,record run, record gym routine, etc.), Internet of Things (IOT) actionlibrary (e.g., switch lights, set thermostat, lock door, etc.), socialaction library (e.g., post a status, share a photo, etc.), communicationaction library (e.g., text Mom, email Bob, etc.), and payment actionlibrary (e.g., pay with card, redeem rewards, etc.).

The processing engine 206 receives the sensor data associated with thesensors from the mobile computing device 115 via the signal line 365,processes the sensor data, filters the sensor data and sends thefiltered data to the context determiner 208 via the signal line 377. Thecontext determiner 208 receives the filtered data from the processingengine 206, combines different types of data in the filtered data,determines a context using one or more activity recognition algorithms,and sends the context determination to the action recommender 210 viathe signal line 375. The context determiner 208 is coupled to the datastorage 243 via the signal line 363.

Turning now to the action recommender 210, the action recommender 210receives information including the action library from the actionlibrary curator 204 and the context from the context determiner 208. Insome embodiments, the action recommender 210 creates action models forturning predictions which can be used to surface the actionable itemsfrom the action library 224. The action recommender 210 stores theaction models in the data storage 243. The action recommender 210 may becoupled to the data storage 243 via signal line 361. The actionrecommender 210 sends instructions to the user interface engine 212 todisplay the actionable items in the user interface of the mobilecomputing device 115 via the signal line 355. The user interface engine212 receives user selection of an actionable item from the mobilecomputing device 115 via the signal line 367 and transmits the selectionto the action recommender 210. The action recommender 210 cooperateswith the API 109 to perform the function associated with the actionableitem.

As shown in the example of FIG. 4, the graphical representationsillustrate user interfaces 400 a-400 c for surfacing actionable itemsrelating to an application in the application launcher. The userinterface 400 a of a mobile computing device 401 includes twocomponents: a system bar 403 and an application launcher 405. The systembar 403 depicts icons and/or graphics relating to clock, cell service,battery power, antenna signals, etc. that persists on the display. Theapplication launcher 405 depicts a set of icons of applications stackedvertically. The user may scroll the application launcher 405 up and downto view more icons. The user interface 400 b depicts a prolonged userselection of the music application icon 407 in the application launcher405. For example, the prolonged user selection can be a long-pressgesture where the icon 407 is touched and pressed for a few seconds bythe user without sliding a finger. The black solid circle 413 isrepresentative of the gesture of prolonged user selection with respectto the icon 407. The user interface 400 c depicts an action menu 409. Inresponse to receiving the long-press gesture, the action recommender 210generates the action menu 409 relating to music application 407. Theaction menu 409 includes a list 411 of actionable items associated withthe music application 407 retrieved from the action library 224 in thedata storage 243. The list 411 of actionable items is tailored tosurface the most commonly accessed content items and/or recentlyaccessed content items in the action menu of the music application 407.In the context of the music application 407, the action menu 409provides deep links to a limited list 411 of functions that can beinitiated without launching the music application 407. In someembodiments, the action menu 409 can be generated as a translucentoverlay over the application launcher 405 blurring the application iconsin the background. In some embodiments, the action recommender 210receives information including a contextual state during which thelong-press gesture of the application is received. The actionrecommender 210 uses the contextual state to surface tailored actionableitems in the action menu of the application. For example, the actionrecommender 210 determines a context of the user is indicative of theuser being home. In response to the user uses applying a long-pressgesture on the music application 407, the action recommender 210determines a tailored list of actionable items that include playlistselections that are relaxing as opposed to high energy playlistselections for a fitness activity.

FIG. 5 shows example graphical representations illustrating embodimentsof user interfaces 500 a-500 c depicting an action menu relating to anapplication. The user interface 500 a depicts a first embodiment of anaction menu 409 that includes a list 513 of system actions related tothe music application 407. If there are no actionable items availablefor an application, in some embodiments, the action recommender 210generates the action menu 409 with the list 513 of system actions of themusic application 407 expanded in a default state. The list 513 ofsystem actions include application information, move icon command, andan uninstall command. The system action is also a form of actionableitem that can be invoked to perform the associated function. The userinterface 500 b depicts a second embodiment of an action menu 409 thatincludes a combination of lists 411 and 513 related to the musicapplication 407. The user interface 500 c depicts a third embodiment ofan action menu 409 that includes a list 515 of unread notificationsrelating to the music application 407 in addition to the list 411 ofactionable items. In some embodiments, the action recommender 210identifies the unread notifications as important set of actions andplaces the unread notifications at the top of the action menu 409. Thelist 515 of unread notifications visually indicates a count of theunread notifications. The unread notifications can be expanded for afull view without needing to launch the music application 407. When thenotifications in the list 515 is dismissed from the action menu 409, itis also dismissed from the notification bar and vice versa.

FIG. 6 shows example graphical representations illustrating userinterfaces 600 a-600 d for editing the actions in the action menu. Theuser interface 600 a depicts an action menu 409 that includes an editbutton 617 for adding or removing the actionable items related to themusic application 407. The user interface 600 b depicts an action menu611 that is updated in response to the user selection of the edit button617 in the user interface 600 a. The action menu 611 includes a removebutton 623 for each actionable item and an add button 619 to addactionable items. The user interface 600 c depicts an action menu 613that is updated in response to the user selection of the add button 619in the user interface 600 b. The action menu 613 includes a set ofactionable items and the user may add an actionable item by selectingthe add button 621 next to each actionable item. The user interface 600d depicts a screen to confirm changes to the action menu. The user canselect save button 625 to save the changes.

In some embodiments, the action recommender 210 surfaces actionableitems relating to a second application in the action menu of a firstapplication. As shown in the example of FIG. 7, the example graphicalrepresentation illustrates user interfaces 700 a-700 d for surfacingactionable items relating to a second application in the action menu ofa first application. The user interface 700 a depicts an applicationlauncher 701 including a set of application icons. The user interface700 b depicts a prolonged user selection of the messaging applicationicon 703. The black solid circle 711 is representative of the gesture ofprolonged user selection with respect to the icon 703. The userinterface 700 c depicts an action menu 709 relating to the messagingapplication 703. In response to receiving the gesture, the actionrecommender 210 generates the action menu 709 relating to messagingapplication 703. The action menu 709 includes a cross-pollinated list705 of actionable items from other applications in addition to theactionable items from the messaging application 703. For example, theactionable items 713 and 715 belong to the other applications in thecommunication category. The user interface element of each actionableitem includes an associated application icon. The user interface 700 ddepicts a secondary action screen 707 to complete the action selected inthe user interface 700 c. In response to the user selecting theactionable item 717 in the user interface 700 c, the action recommender210 generates the secondary action screen 707 where the user may enterthe message to send to the selected contact without launching themessaging application 703.

In some embodiments, the action recommender 210 surfaces actionableitems from the action library 224 relating to a notification of anapplication. As shown in the example of FIG. 8, the graphicalrepresentation illustrates user interfaces 800 a-800 c for surfacingactionable items relating to the notification of an application in theuser interface. The user interface 800 a depicts an incomingnotification 801 from a calendaring application in the applicationlauncher 803. The user interface 800 b depicts an action menu 805relating to the notification 801. The action menu 805 includes a list807 of actionable items. In response to the user selecting thenotification 801 in the user interface 800 a, the action recommender 210generates the action menu 805 relating to the notification 801 of thecalendaring application and presents the action menu 805 as a drop downoverlay on top of the application launcher 803. The user interface 800 cdepicts a secondary action screen 811 to complete the action selected inthe user interface 800 b. In response to the user selecting theactionable item 809 in the action menu 805 in the user interface 800 b,the action recommender 210 generates the secondary action screen 811where the user may request a rideshare car service without launching theridesharing application.

In some embodiments, the action recommender 210 surfaces actionableitems from the action library 224 in response to a gesture input. Asshown in the example of FIG. 9, the graphical representations illustrateuser interfaces 900 a-900 d for surfacing actionable items in responseto a gesture input. The user interface 900 a depicts the applicationlauncher 901 including a set of application icons. The user interface900 b depicts a gesture pad 903 brought into view in response to theuser swiping up the display from the bottom of the user interface 900 a.In some embodiments, the gesture pad 903 may be brought into view from alock screen. The gesture pad 903 includes four application icons 909,911, 913 and 915. The action menu relating to these four applicationscan be surfaced from the gesture pad 903 by the user, if needed. Theuser interface 900 c depicts a stroke input 905 of a letter ‘S’ on thegesture pad 903 by the user. The action recommender 210 surfacesactionable items contextual to the stroke input 905. For example, asshown in the user interface 900 d, the action recommender 210 generatesa list 907 of actionable items relating to the letter ‘S’. Theactionable items may include a shortcut to applications starting withthe letter ‘S’.

In some embodiments, the action recommender 210 surfaces actionableitems from the action library 224 in response to a contextual trigger ofan activity detected by the context determiner 208. As shown in theexample of FIG. 10, the graphical representations illustrate userinterfaces 1000 a-1000 d for surfacing actionable items in response to acontextual activity. The user interface 1000 a depicts a firstembodiment in which a car of a user is detected in proximity of themobile computing device 115. The user interface 1000 a depicts anotification 1001 of the car detected nearby. The notification 1000 a isan overlay over the application launcher window. The user interface 1000a includes a list 1003 of actionable items that is contextually relatedto the car detected in proximity. For example, the user may select theactionable items in the list 1003 to unlock the car, start the engineand/or open the trunk. These actionable items may be associated with avehicle application enabling the mobile computing device 115 to remotelycontrol the car. The remote control functions included in the list 1003are performed without the user having to launch the vehicle application.The user may remove the notification by selecting the cancel button1005. The user interface 1000 b depicts a second embodiment in which theuser location inside a coffeehouse is detected. The user interface 1000b depicts a notification 1007 inquiring if the user is in thecoffeehouse and includes a list 1009 of actionable items contextuallyrelated to the user detected in the coffeehouse. For example, the usermay select the actionable items in the list 1009 to pay for coffee witha card, check in on the social network, and/or redeem rewards usingloyalty points. The user interface 1000 c depicts a third embodiment inwhich user engaging in a fitness activity is detected. The userinterface 1000 c depicts a notification 1013 inquiring if the user isstarting an exercise and includes a list 1015 of actionable itemsrelating to the context of the fitness activity. For example, the usermay select the actionable items in the list 1015 to start recording thecycling activity, start recording a run or start recording a walk. Theuser interface 1000 d depicts a fourth embodiment in which user isdetected to be commuting. The user interface 1000 d depicts anotification 1019 inquiring if the user is commuting and includes a list1017 of actionable items relating to the context of commuting. Forexample, the user may select the actionable items in the list 1017 toget directions to the destination, stream a playlist while commutingand/or view the traffic conditions.

FIGS. 11A-11B are example graphical representations of embodiments ofthe user interface for customizing settings on the mobile computingdevice 115. In FIG. 11A, the graphical representation illustrates anembodiment of a user interface 1100 for toggling an improved form ofairplane mode. The user interface 1100 includes an Airplane Plus mode1101 that suspends radio-frequency signal transmission including but notlimited to telephony, Bluetooth, Wi-Fi, GPS, etc. when the mobilecomputing device 115 is switched to a sleep or standby mode. Forexample, the mobile computing device 115 is put to sleep when thesleep/wake physical button that turns a display of the mobile computingdevice 115 on and off is pressed. The radio-frequency signaltransmission is activated when the mobile computing device 115 is wokenup. The Airplane Plus mode 1101 includes exceptions for applicationsthat are currently running on the mobile computing device 115. Forexample, a phone call in progress is not disrupted when the mobilecomputing device 115 goes to standby mode after a few minutes into thephone call. Likewise, a navigation via web mapping application is notdisrupted in a similar example. The Airplane Plus mode 1101 is toggledusing the button 1103. In FIG. 11B, the graphical representationillustrates an embodiment of a user interface 1110 for toggling animproved form of Do Not Disturb mode. The user interface 1110 includes aLife Mode 1107 that limits interruptions from background applicationsand notifications on the mobile computing device 115. The user interface1110 includes a section for selecting exceptions where the user mayselect which applications 1105 can notify the user while in Life Mode1107.

FIG. 12 is a flow diagram illustrating one embodiment of an examplemethod 1200 for creating an action library. At 1202, the archiver 202identifies an application and a parameter associated with theapplication. The archiver 202 archives user interaction withapplications on the mobile computing device 115. For example, thearchiver 202 identifies that the user launched an email application,selected a contact named “Bob” in the address list, and sent an emailmessage to the contact. At 1204, the action library curator 204determines a function of the application accessible via an applicationprogramming interface (API) associated with the application. The actionlibrary curator 204 uses an API key to access the API 109 of the onlineservice 111 associated with the application as shown in FIG. 1. If theAPI 109 accepts the API key as valid, the action library curator 204uses an API to parse an application and determines its constituentfunctions, tasks, or activities. For example, the action library curator204 determines that the constituent functions of an email applicationinclude create a new email message, set up an appointment, create acalendar invite, etc. At 1206, the action library curator 204 determinesan application programming interface (API) call associated withperforming the function. The API call can be used submit a request tothe API 109 to perform a function of the application. The API 109services the request of the API call. The request includes theparameters or data associated with performing the function. At 1208, theaction library curator 204 generates an actionable item based on the APIcall and the parameter. The actionable item of an application may be auser interface element that links directly to an API call toautomatically transmit a request to the API 109 of the online service111 for performing a function, task or activity associated with theapplication. At 1210, the action library curator 204 classifies acategory of the actionable item. For example, the action library curator204 identifies actionable items for one or more fitness applications andgroups the identified actionable items under a ‘fitness’ category. At1212, the action library curator 204 stores the actionable item and thecategory in an action library.

FIG. 13 is a flow diagram illustrating one embodiment of an examplemethod 1300 for generating an action menu relating to an application. At1302, the processing engine 206 receives a prolonged selection of anapplication in association with an application launcher. At 1304, thecontext determiner 208 determines a context associated with theprolonged selection of the application. At 1306, the action recommender210 determines an action library including a plurality of actionableitems associated with the application. At 1308, the action recommender210 determines an actionable item associated with the application fromthe action library based on the context. At 1310, the action recommender210 generates an action menu including a direct link to perform theactionable item. For example, the function of the actionable item can beperformed without launching the corresponding application. At 1312, theuser interface engine 212 presents the action menu associated with theapplication in response to the prolonged selection.

FIG. 14 is a flow diagram illustrating one embodiment of an examplemethod 1400 for surfacing actionable items relating to a secondapplication in the action menu of a first application (cross-pollinationof actions). At 1402, the processing engine 206 receives a prolongedselection of a first application in association with an applicationlauncher. At 1404, the action recommender 210 determines a category ofthe first application. At 1406, the action recommender 210 determines anaction library including a plurality of actionable items associated withthe category. At 1408, the action recommender 210 determines anactionable item associated with a second application from the actionlibrary. At 1410, the action recommender 210 generates an action menuincluding a direct link to perform the actionable item associated withthe second application. At 1412, the user interface engine 212 presentsthe action menu in association with the first application in response tothe prolonged selection.

FIG. 15 is a flow diagram illustrating one embodiment of an examplemethod 1500 for surfacing actionable items relating to a notification ofan application in an action menu. At 1502, the processing engine 206receives a notification associated with a first application. At 1504,the context determiner 208 determines a context associated with thenotification. At 1506, the action recommender 210 determines an actionlibrary including a plurality of actionable items. At 1508, the actionrecommender 210 determines an actionable item associated with a secondapplication from the action library based on the context. At 1510, theaction recommender 210 generates an action menu including a direct linkto perform the actionable item. At 1512, the user interface engine 212presents the action menu in association with the notification.

FIG. 16 is a flow diagram illustrating one embodiment of an examplemethod 1600 for surfacing actionable items relating to a detectedactivity and associated context. At 1602, the processing engine 206receives a signal stream from a mobile computing device. At 1604, theprocessing engine 206 processes the signal stream. At 1606, the contextdeterminer 208 detects an activity and a context of the activity basedon the processed signal stream. At 1608, the action recommender 210determines an action library including a plurality of actionable itemsassociated with one or more applications. At 1610, the actionrecommender 210 determines an actionable item relating to the activityand the context of the activity from the action library. At 1612, theaction recommender 210 generates an action menu including a direct linkto perform the actionable item. At 1614, the user interface engine 212presents the action menu in association with the detected activity.

A system and method for surfacing tailored actionable items associatedwith applications in a user experience of a mobile computing device hasbeen described. In the above description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the techniques introduced above. It will be apparent,however, to one skilled in the art that the techniques can be practicedwithout these specific details. In other instances, structures anddevices are shown in block diagram form in order to avoid obscuring thedescription and for ease of understanding. For example, the techniquesare described in one embodiment above primarily with reference tosoftware and particular hardware. However, the present invention appliesto any type of computing system that can receive data and commands, andpresent information as part of any peripheral devices providingservices.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some portions of the detailed descriptions described above are presentedin terms of algorithms and symbolic representations of operations ondata bits within a computer memory. These algorithmic descriptions andrepresentations are, in some circumstances, used by those skilled in thedata processing arts to convey the substance of their work to othersskilled in the art. An algorithm is here, and generally, conceived to bea self-consistent sequence of steps leading to a desired result. Thesteps are those requiring physical manipulations of physical quantities.Usually, though not necessarily, these quantities take the form ofelectrical or magnetic signals capable of being stored, transferred,combined, compared, and otherwise manipulated. It has proven convenientat times, principally for reasons of common usage, to refer to thesesignals as bits, values, elements, symbols, characters, terms, numbers,or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing”, “computing”, “calculating”, “determining”,“displaying”, or the like, refer to the action and processes of acomputer system, or similar electronic computing device, thatmanipulates and transforms data represented as physical (electronic)quantities within the computer system's registers and memories intoother data similarly represented as physical quantities within thecomputer system memories or registers or other such information storage,transmission or display devices.

The techniques also relate to an apparatus for performing the operationsherein. This apparatus may be specially constructed for the requiredpurposes, or it may comprise a general-purpose computer selectivelyactivated or reconfigured by a computer program stored in the computer.Such a computer program may be stored in a non-transitory computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, and magnetic disks,read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, flash memories including USB keyswith non-volatile memory or any type of media suitable for storingelectronic instructions, each coupled to a computer system bus.

Some embodiments can take the form of an entirely hardware embodiment,an entirely software embodiment or an embodiment containing bothhardware and software elements. One embodiment is implemented insoftware, which includes but is not limited to firmware, residentsoftware, microcode, etc.

Furthermore, some embodiments can take the form of a computer programproduct accessible from a computer-usable or computer-readable mediumproviding program code for use by or in connection with a computer orany instruction execution system. For the purposes of this description,a computer-usable or computer readable medium can be any apparatus thatcan contain, store, communicate, propagate, or transport the program foruse by or in connection with the instruction execution system,apparatus, or device.

A data processing system suitable for storing and/or executing programcode can include at least one processor coupled directly or indirectlyto memory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers.

Network adapters may also be coupled to the system to enable the dataprocessing system to become coupled to other data processing systems orremote printers or storage devices through intervening private or publicnetworks. Modems, cable modem and Ethernet cards are just a few of thecurrently available types of network adapters.

Finally, the algorithms and displays presented herein are not inherentlyrelated to any particular computer or other apparatus. Variousgeneral-purpose systems may be used with programs in accordance with theteachings herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps. The requiredstructure for a variety of these systems will appear from thedescription above. In addition, the techniques are not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the various embodiments as described herein.

The foregoing description of the embodiments has been presented for thepurposes of illustration and description. It is not intended to beexhaustive or to limit the specification to the precise form disclosed.Many modifications and variations are possible in light of the aboveteaching. It is intended that the scope of the embodiments be limitednot by this detailed description, but rather by the claims of thisapplication. As will be understood by those familiar with the art, theexamples may be embodied in other specific forms without departing fromthe spirit or essential characteristics thereof. Likewise, theparticular naming and division of the modules, routines, features,attributes, methodologies and other aspects are not mandatory orsignificant, and the mechanisms that implement the description or itsfeatures may have different names, divisions and/or formats.Furthermore, as will be apparent to one of ordinary skill in therelevant art, the modules, routines, features, attributes, methodologiesand other aspects of the specification can be implemented as software,hardware, firmware or any combination of the three. Also, wherever acomponent, an example of which is a module, of the specification isimplemented as software, the component can be implemented as astandalone program, as part of a larger program, as a plurality ofseparate programs, as a statically or dynamically linked library, as akernel loadable module, as a device driver, and/or in every and anyother way known now or in the future to those of ordinary skill in theart of computer programming. Additionally, the specification is in noway limited to embodiment in any specific programming language, or forany specific operating system or environment. Accordingly, thedisclosure is intended to be illustrative, but not limiting, of thescope of the specification, which is set forth in the following claims.

What is claimed is:
 1. A computer-implemented method comprising:receiving, using one or more processors, sensor data and application usedata from a client device; analyzing, using the one or more processors,the sensor data and the application use data to determine a contextualstate of use of the client device; determining, using the one or moreprocessors, whether the contextual state of use of the client device ispersisting for a threshold period of time when a first application isselected by a user on the client device; responsive to determining thatthe contextual state of use of the client device is persisting for thethreshold period of time when the first application is selected by theuser on the client device, identifying, using the one or moreprocessors, from an action library including a plurality of actionableitems, a first actionable item of the first application using thecontextual state of use as input to an action machine learning model,the action machine learning model being trained using a datasetcorresponding to a demographic group to which the user belongs;generating, using the one or more processors, a first action menu forthe first application using the first actionable item responsive toidentifying the first actionable item, the first actionable itemincluding a first deep link to perform an action within the firstapplication; presenting, using the one or more processors, the firstaction menu in association with the first application while the firstapplication is remaining unlaunched on the client device; receiving,using the one or more processors, user selection of the first actionableitem in the first action menu; and responsive to receiving the userselection of the first actionable item, linking directly from the firstaction menu and past an interstitial page in the first application to atarget page corresponding to performing the action on the client device.2. The computer-implemented method of claim 1, further comprisingdetermining the action library by: identifying an application and aparameter associated with the application; connecting with anapplication programming interface associated with the application;determining a function of the application accessible via the applicationprogramming interface; determining an application programming interfacecall associated with performing the function; generating an actionableitem for the application using the application programming interfacecall and the parameter; classifying a category of the actionable item;and storing the actionable item and the category of the actionable itemin the action library.
 3. The computer-implemented method of claim 1,wherein the first application is selected by the user using a prolongedselection of the first application in an application launcher andgenerating the first action menu further comprises: determining acategory of the first application; identifying, from the action library,a second actionable item of a second application based on the categoryof the first application matching with the second actionable item;adding the second actionable item to the first action menu for the firstapplication, the second actionable item including a second deep link toperform an action within the second application; and presenting thefirst action menu in association with the first application in responseto the prolonged selection.
 4. The computer-implemented method of claim1, further comprising: detecting an incoming notification for the firstapplication on the client device; determining a context of the incomingnotification; identifying, from the action library, a second actionableitem of a second application based on the context; generating a secondaction menu corresponding to the incoming notification using the secondactionable item, the second actionable item including a second deep linkto perform an action within the second application; and presenting thesecond action menu in association with the incoming notification.
 5. Thecomputer-implemented method of claim 1, wherein the client device is amobile computing device and further comprising: receiving a signalstream from the mobile computing device; processing the signal stream;detecting a user activity and a context of the user activity based onthe processed signal stream; identifying, from the action library, athird actionable item of a third application based on the context; andgenerating a third action menu corresponding to the context of the useractivity using the third actionable item, the third actionable itemincluding a third deep link to perform an action within the thirdapplication.
 6. The computer-implemented method of claim 5, furthercomprising: determining whether a duration of the detected user activitysatisfies a predetermined threshold; and responsive to determining thatthe duration of the detected user activity satisfies the predeterminedthreshold, generating a notification to surface the third action menuand presenting the notification in association with the detected useractivity.
 7. The computer-implemented method of claim 5, whereingenerating the third action menu further comprises: determining whetherthe third application is loaded on the mobile computing device; andresponsive to determining that the third application is unloaded on themobile computing device, generating the third action menu to include afourth link to load the third application on the mobile computingdevice.
 8. The computer-implemented method of claim 1, furthercomprising: responsive to receiving the user selection of the firstactionable item, generating a secondary action screen for the firstapplication and linking directly from the first action menu to thesecondary action screen; presenting the secondary action screen on theclient device; receiving user input in the secondary action screen tocomplete performing the action within the first application representedby the first actionable item; and performing the action of the firstactionable item based on the user input.
 9. The computer-implementedmethod of claim 1, wherein linking directly from the first action menuto the target page corresponding to performing the action skips atraversal of the interstitial page in the first application on theclient device.
 10. A system comprising: one or more processors; and amemory, the memory storing instructions, which when executed cause theone or more processors to: receive sensor data and application use datafrom a client device; analyze the sensor data and the application usedata to determine a contextual state of use of the client device;determine whether the contextual state of use of the client device ispersisting for a threshold period of time when a first application isselected by a user on the client device; responsive to determining thatthe contextual state of use of the client device is persisting for thethreshold period of time when the first application is selected by theuser on the client device, identify, from an action library including aplurality of actionable items, a first actionable item of the firstapplication using the contextual state of use as input to an actionmachine learning model, the action machine learning model being trainedusing a dataset corresponding to a demographic group to which the userbelongs; generate a first action menu for the first application usingthe first actionable item responsive to identifying the first actionableitem, the first actionable item including a first deep link to performan action within the first application; present the first action menu inassociation with the first application while the first application isremaining unlaunched on the client device; receive user selection of thefirst actionable item in the first action menu; and responsive toreceiving the user selection of the first actionable item, link directlyfrom the first action menu and past an interstitial page in the firstapplication to a target page corresponding to performing the action onthe client device.
 11. The system of claim 10, wherein the instructionsfurther cause the one or more processors to determine the actionlibrary, wherein the one or more processors are configured to: identifyan application and a parameter associated with the application; connectwith an application programming interface associated with theapplication; determine a function of the application accessible via theapplication programming interface; determine an application programminginterface call associated with performing the function; generate anactionable item for the application using the application programminginterface call and the parameter; classify a category of the actionableitem; and store the actionable item and the category of the actionableitem in the action library.
 12. The system of claim 10, wherein togenerate the first action menu, the instructions further cause the oneor more processors to: determine a category of the first application;identify, from the action library, a second actionable item of a secondapplication based on the category of the first application matching withthe second actionable item; add the second actionable item to the firstaction menu for the first application, the second actionable itemincluding a second deep link to perform an action within the secondapplication; and present the first action menu in association with thefirst application in response to an event, wherein the event is aprolonged selection of the first application in an application launcher.13. The system of claim 10, wherein the instructions further cause theone or more processors to: detect an incoming notification for the firstapplication on the client device; determine a context of the incomingnotification; identify, from the action library, a second actionableitem of a second application based on the context; generate a secondaction menu corresponding to the incoming notification using the secondactionable item, the second actionable item including a second deep linkto perform an action within the second application; and present thesecond action menu in association with the incoming notification. 14.The system of claim 10, wherein the client device is a mobile computingdevice and the instructions further cause the one or more processors to:receive a signal stream from the mobile computing device; process thesignal stream; detect a user activity and a context of the user activitybased on the processed signal stream; identify, from the action library,a third actionable item of a third application based on the context; andgenerate a third action menu corresponding to the context of the useractivity using the third actionable item, the third actionable itemincluding a third deep link to perform an action within the thirdapplication.
 15. The system of claim 14, wherein the instructionsfurther cause the one or more processors to: determine whether aduration of the detected user activity satisfies a predeterminedthreshold; and responsive to determining that the duration of thedetected user activity satisfies the predetermined threshold, generate anotification to surface the third action menu and present thenotification in association with the detected user activity.
 16. Thesystem of claim 14, wherein to generate the third action menu, theinstructions further cause the one or more processors to: determinewhether the third application is loaded on the mobile computing device;and responsive to determining that the third application is unloaded onthe mobile computing device, generate the third action menu to include afourth link to load the third application on the mobile computingdevice.
 17. The system of claim 10, wherein the instructions furthercause the one or more processors to: responsive to receiving the userselection of the first actionable item, generate a secondary actionscreen for the first application and linking directly from the firstaction menu to the secondary action screen; present the secondary actionscreen on the client device; receive user input in the secondary actionscreen to complete performing the action within the first applicationrepresented by the first actionable item; and perform the action of thefirst actionable item based on the user input.
 18. The system of claim10, wherein linking directly from the first action menu to the targetpage corresponding to performing the action skips a traversal of theinterstitial page in the first application on the client device.
 19. Acomputer-implemented method comprising: receiving, using one or moreprocessors, sensor data and application use data from a mobile computingdevice of a user; analyzing, using the one or more processors, thesensor data and the application use data to determine an activity of theuser and a context of the activity; determining, using the one or moreprocessors, whether the context of the activity is persisting for athreshold period of time; responsive to determining that the context ofthe activity is persisting for the threshold period of time,determining, using the one or more processors, an actionable item usingthe activity of the user and the context of the activity as input to anaction machine learning model and an action library including aplurality of actionable items of one or more applications, the actionmachine learning model being trained using a dataset corresponding to ademographic group to which the user belongs; generating, using the oneor more processors, an action menu using the actionable item, theactionable item including a direct link to perform an action within anapplication represented by the actionable item; presenting, using theone or more processors, the action menu in association with thedetermined activity while the application is remaining unlaunched on themobile computing device; receiving, using the one or more processors, aselection by the user of the actionable item in the action menu; andresponsive to receiving the selection by the user of the actionable itemin the action menu, linking directly from the action menu and past aninterstitial page in the application to a target page corresponding toperforming the action on the mobile computing device.
 20. Acomputer-implemented method comprising: receiving, using one or moreprocessors, sensor data and application use data from a client device;analyzing, using the one or more processors, the sensor data and theapplication use data to determine a contextual state of use of theclient device; determining, using the one or more processors, whetherthe contextual state of use of the client device is persisting for athreshold period of time when a prolonged selection of a firstapplication from a user is received on an application launcher on theclient device; responsive to determining that the contextual state ofuse of the client device is persisting for the threshold period of time,determining, using the one or more processors, a first actionable itemof the first application and a second actionable item of a secondapplication using the contextual state of use as input to an actionmachine learning model and an action library including a plurality ofactionable items, the action machine learning model being trained usinga dataset corresponding to a demographic group to which the user belongsand the first application and the second application corresponding to asame category; generating, using the one or more processors, an actionmenu for the first application using the first actionable item and thesecond actionable item responsive to determining the first actionableitem and the second actionable item, the first actionable item includinga first deep link to perform a first action within the first applicationand the second actionable item including a second deep link to perform asecond action within the second application; presenting, using the oneor more processors, the action menu in association with the firstapplication while the first application and the second application areremaining unlaunched on the application launcher of the client device;receiving, using the one or more processors, user selection of one ofthe first actionable item and the second actionable item in the actionmenu; and responsive to receiving the user selection of one of the firstactionable item and the second actionable item, linking directly fromthe action menu and past an interstitial page in one of the firstapplication and the second application to a target page corresponding toperforming one of the first action and the second action on the clientdevice.